Device and Method of Measuring Data for Calibration, 
Program for Measuring Data for Calibration, Program 
Recording Medium Readable with Computer, and Image Data 
Processing Device 

Background of the Invention 
[0001] 1. Field of the Invention 

[0002] The present invention relate to a device and a 
method for measuring data for calibration used when a lens 
aberration in a photographed image is corrected or when a 
lens aberration-corrected image for use in stereo image 
measurement or the like is necessary and, more 
particularly, to a device and a method for measuring data 
for calibration with which the internal parameters of a 
camera, such as the principal point position of the lens, 
screen distance (focal length) and distortion parameters, 
necessary to obtain a lens aberration corrected-image can 
be obtained with ease even if the camera is of the type in 
which optical conditions can be varied (a zoom camera, for 
example) . 

[0003] The present invention also relates to an image 
processing device capable of removing the effect of lens 
aberration in a stereo image photographed with a camera of 
the type in which optical conditions can be varied (a zoom 
camera, for example) to allow precise three-dimensional 
measurement of the shape of an object by stereo image 
measurement. 

2. Description of the Related Art 

[0004] Conventionally, it is important to obtain an image 
with low aberration in the fields of photogrammetry and 
photographic measurement. In the fields of photogrammetry 
and photographic measurement, high-accuracy lenses with a 
low aberration are therefore used. In the field of 
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photogrammetry, the internal parameters of a camera 
(principal point position, screen distance, and distortion 
parameters) are analytically obtained in a way that a 
multiplicity of points measured precisely are positioned in 
a three-dimensional space and then measured from various 
angles. In the case of a measuring camera for use in the 
field of photographic measurement, the internal parameters 
of a camera are obtained by measuring precisely a 
fabricated camera. 

[0005] However, an image photographed with a commercially 
available digital camera cannot be used in stereo image 
measurement of an object without correction because the 
lens distortion is large. Thus, it is necessary to correct 
the image using the internal parameters of the digital 
camera (principal point position, screen distance and lens 
distortion) and to measure the focal length of the digital 
camera at which the image was taken precisely to enhance 
the three-dimensional precision. Description will be 
hereinafter made of a fixed focus camera and a multi-focus 
camera separately. 

[0006] Conventionally, camera calibration is performed on a 
fixed focus stereo camera and a photograph of an object for 
stereo image measurement is taken at its fixed focal point. 
However, a fixed focus stereo camera cannot adjust the 
focal length depending upon the objects, so that the image 
may be out of focus or the photographed object may not be 
large enough to fill the image area under some 
photographing conditions. Thus, it is difficult to obtain 
a stereo image of quality sufficient to be used in stereo 
image measurement. In addition, when the camera or the 
lens is changed depending upon the size of the object or 
the necessary precision to select a camera or a lens with a 
proper focal length, calibration must be performed again on 
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the selected camera. Thus, three-dimensional measurement 
cannot be easily performed with a fixed focus stereo 
camera, which prevents spreading of stereo image 
measurement . 

[0007] In a multi-focus digital camera employing a so- 
called zoom lens, the lens distortion is large, and the 
depth accuracy and the lens distortion vary with the focal 
length. Thus, when the focal length is changed, 
calibration must be performed again. It is, therefore, 
difficult to carry out image measurement using a zoom lens 
from the viewpoint of work efficiency. Thus, even a multi- 
focus camera is used as a fixed focus camera without using 
the function of the zoom lens, which makes stereo image 
measurement on site inconvenient. 

Summary of the Invention 
[0008] The present invention has been made to solve the 
above problems, and a first object of the present invention 
is to provide a device and a method for measuring data for 
calibration with which the internal parameters of a camera 
necessary to obtain a high quality image free from the 
effect of lens distortion can be obtained with ease even if 
the camera is of the type in which optical conditions can 
be varied (a zoom camera, for example) . 
[0009] A second object of the present invention is to 
provide an image processing device capable of removing the 
effect of lens aberration in a stereo image photographed 
with a camera of the type in which optical conditions can 
be varied (a zoom camera, for example) to allow precise 
three-dimensional measurement of the shape of an object in 
stereo image measurement. 

[0010] The inventive device for measuring data for 
calibration, which achieves the first object, is for a 
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purpose of obtaining data for calibration of a camera 2 
capable of varying its optical conditions, using a 
plurality of the images of a calibration chart 1 having 
marks arranged thereon which were photographed with the 
camera 2 under varied optical conditions, and comprises, as 
shown in FIG. 1, a mark extracting part (131, 132, 133) for 
extracting the marks from the images of the chart; an 
internal parameter calculating part 134 for calculating 
data for calibration under the optical conditions under 
which the images of the chart were photographed based on 
the positions of the marks extracted by the mark extracting 
part and a plurality of conditions under which the images 
of the chart were photographed; and an internal parameter 
function calculating part 160 for calculating data for 
calibration corresponding to the varied optical conditions 
in the camera 2 in photographing using the data for 
calibration calculated in the internal parameter 
calculating part 134 and a plurality of optical conditions 
under which the images of the chart were photographed. 

[0011] A camera capable of varying its optical conditions 
herein is a camera with a zoom lens in which the focal 
length can be changed or an auto-focus camera in which the 
lens can be adjusted to adjust the focus. The images of a 
chart photographed under varied optical conditions are 
images of a chart photographed at various focal lengths or 
at various lens positions using an auto-focus mechanism of 
a camera. The varied optical conditions in the camera in 
photographing are the individual focal lengths adjusted 
with a zoom lens or the like. The images of the chart are 
photographed at the focal length. The data for calibration 
include at least either of data of the principal point of a 
lens or data of the distortion aberration of the lens. 

[0012] In the device constituted as above, the mark 
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extracting part extracts the marks from the images of the 
chart and, preferably, can extract the positions of the 
marks. The internal parameter calculating part 134 
calculates data for calibration under the optical 
conditions under which the images of the chart were 
photographed based on the positions of the marks extracted 
by the mark extracting part and, preferably, uses a group 
of images of the chart photographed under the same optical 
condition as the base of the calculation. The internal 
parameter function calculating part 160 calculates data for 
calibration corresponding to the varied optical conditions 
of the camera 2 in photographing using the data for 
calibration for each of the optical photographing 
conditions calculated in the internal parameter calculating 
part 134 and a plurality of optical conditions under which 
the images of the chart were photographed. Preferably, the 
internal parameter calculating part 160 has mathematical 
expressions for obtaining function forms of coefficients 
constituting the internal parameter functions and for 
calculating the internal parameters using coefficients 
corresponding to the varied optical photographing 
conditions of the camera 2. 

[0013] Preferably, the chart for calibration is flat as 
shown in FIG. 1, and the plurality of the images of the 
chart are stereo images photographed in such a manner that 
data for calibration can be calculated therefrom by the 
mark extracting part (131, 132, 133) and the internal 
parameter calculating part 134 as shown in FIG. 5 to FIG. 
7. 

[0014] Preferably, the chart for calibration is three- 
dimensional (three-dimensional reference chart 20) as shown 
in FIG. 14, and the plurality of images of the chart are 
stereo images photographed in such a manner that data for 
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calibration can be calculated therefrom by a mark 
extracting parts 171 and an internal parameter calculating 
part 173. 

[0015] Preferably, the data for calibration calculated in 
the internal parameter calculating part 160 can be 
continuously calculated according to the varied optical 
photographing conditions of the camera as shown in FIG. 13. 
[0016] The method for measuring data for calibration of the 
present invention, which achieves the first object, 
comprises: a step of photographing a plurality of images of 
a calibration chart having marks arranged thereon, with a 
camera capable of varying its optical conditions under 
varied optical conditions (S102, S104); a mark extracting 
step for extracting the marks from the images of the chart 
(S105) ; an internal parameter calculating step for 
calculating data for calibration under which the images of 
the chart were photographed based on the positions of the 
extracted marks (S106) ; and an internal parameter function 
calculating step for calculating data for calibration 
corresponding to the varied optical conditions in the 
camera in photographing using the calculated data for 
calibration and a plurality of optical conditions under 
which the images of the chart were photographed (S110) as 
shown in FIG . 8 . 

[0017] The program for measuring data for calibration of 
the present invention, which achieves the first object, is 
a program used to obtain data for calibration of a camera 
capable of varying its optical conditions, wherein the data 
for calibration are obtained using a plurality of images of 
a calibration having marks arranged thereon which were 
photographed with the camera under varied optical 
conditions, and is for making a computer function as; mark 
extracting means (131, 132, 133) for extracting the marks 
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from the images of the chart; internal parameter 
calculating means 134 for calculating data for calibration 
under optical conditions under which the images of the 
chart were photographed based on the positions of the marks 
extracted by the mark extracting means; and internal 
parameter function calculating means 160 for calculating 
data for calibration corresponding to the varied optical 
photographing conditions of the camera using the data for 
calibration calculated in the internal parameter 
calculating means 134 and a plurality of optical conditions 
under which the images of the chart were photographed. The 
program for measuring data for calibration is stored in the 
computer readable recording medium of the present 
invention. 

[0018] The image data processing device of the present 
invention, which achieves the second object, comprises an 
image data receiving parts (210, 220) for receiving data of 
photographed images and optical photographing conditions 
from a camera 2 capable of varying the optical conditions; 
a calibration data forming part 230 for forming data for 
calibration based on the optical photographing conditions 
received by the image data receiving part; and an image 
processing part 240 for processing the data of photographed 
images based on the data for calibration as shown in FIG. 
15. 

[0019] In the device constituted as above, the image data 
receiving part (210, 220) receives data of photographed 
images and optical photographing conditions from a camera 2 
capable of varying the optical conditions. The calibration 
data forming part 230 forms data for calibration based on 
the optical photographing conditions received by the image 
data receiving part. By the data for calibration, the 
effect of the lens aberration of the camera 2 included in 
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the data of photographed images can be compensated. Since 
the image processing part 240 processes the data of 
photographed images based on the data for calibration, 
precise image data free from the effect of lens aberration 
can be obtained. The image data allows precise three- 
dimensional measurement even when used in measurement of 
height and depth of the peaks and bottoms on a surface 
based on minute parallax differences on an image as stereo 
image measurement. 

[0020] The optical photographing conditions may be directly 
received from the camera 2 as the photograph data receiving 
part 220 does, or may be obtained. indirectly from the data 
of photographed images as the focal length calculating part 
232 does. When the focal length calculating part 232 
calculates the optical photographing conditions, the 
calibration data forming part 230 receives the optical 
photographing conditions from the focal length calculating 
part 232. 

[0021] This application is based on Japanese patent 
application, No. 2002-195058 filed in Japan on July 3, 
2002, which is entirely incorporated herein by reference. 
[0022] The present invention will become more fully 
understood from the detailed description given hereinbelow. 
However, the detailed description and the specific 
embodiment are illustrated of desired embodiments of the 
present invention and are described only for the purpose of 
explanation. Various changes and modifications will be 
apparent to those ordinary skilled in the art on the basis 
of the detailed description. 

[0023] The applicant has no intention to give to public any 
disclosed embodiment. Among the disclosed changes and 
modifications, those which may not literally fall within 
the scope of the patent claims constitute, therefore, a 
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part of the present invention in the sense of doctrine of 
equivalents . 

Brief Description of the Drawings 
[0024] FIG. 1 is a block diagram, showing a device for 
measuring data for calibration according to a first 
embodiment of the present invention; 

[0025] FIG . 2 is a plan view, showing an example of a 
calibration chart; 

[0026] FIGS. 3 (A) -3(C) are explanatory views, illustrating 
examples of a first mark; 

[0027] FIGS. 4 (A) -4(H) are explanatory views, illustrating 
examples of a second mark; 

[0028] FIG. 5 is a perspective view, showing the positions 
of a zoom camera in measuring the lens aberration at 
various focal lengths of the camera; 
[0029] FIGS. 6 (A) -6(H) are views, illustrating the 
procedure of photographing a chart with a zoom- camera and 
showing camera images and the positional relations between 
the chart and the camera corresponding to the camera 
images; 

[0030] FIG. 7 is a view, illustrating a camera distance in 
measuring the lens aberration when the focal length of a 
zoom camera is equivalent to that of a normal lens or a 
telephoto lens; 

[0031] FIG. 8 is a flowchart, explaining a method for 
measuring data for calibration using a device for measuring 
data for calibration; 

[0032] FIG. 9 is a flowchart, explaining the process of 
calculating calibration elements at each focal length; 
[0033] FIG. 10 is a detailed flowchart, explaining the 
process of measuring the precise position of the second 
mark; 
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[0034] FIG. 11 (A) is an explanatory view of an image 
coordinate system and an objective coordinate system in 
center pro j ection; 

[0035] FIG. 11 (B) is an explanatory view of a template 
image for normalized correlation and an objective image for 
use in recognition of targets; 

[0036] FIG. 12 is an explanatory view, illustrating an 
example of a screen which displays the result of the 
operation for obtaining the calibration elements; 
[0037] FIGS. 13 (A) -13(D) are views, illustrating the 
relation between the focal length and the coefficients for 
use in the internal parameter functions; 

[0038] FIG. 14 is a block diagram, illustrating a device 
for measuring data for calibration according to a second 
embodiment of the present invention; 

[0039] FIG. 15 is a block diagram, illustrating an image 
data processing device according to an embodiment of the 
present invention ; 

[0040] FIG. 16 is a view for explaining orientation 
calculation using a model coordinate system XYZ and right 
and left camera coordinate systems xyz; 

[0041] FIG. 17 is a flowchart for explaining the process of 
photographing an object in stereo on site and performing 
stereo image measurement; 

[0042] FIG. 18 shows a stone wall as an example of a field 
for stereo image measurement; 

[0043] FIG. 19 is a view, illustrating photographing 
conditions of a camera; and 

[0044] FIG. 20 is a view, showing the results of experiment 
for measuring the accuracy of stereo image measurement. 

Description of the Preferred Embodiments 
[0045] Description will be hereinafter made of the present 
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invention with reference to the drawings. FIG. 1 is a 
block diagram illustrating a device for measuring data for 
calibration according to a first embodiment of the present 
invention. In FIG. 1, designated as 1 is a chart for 
calibration. The chart 1 is a flat sheet on which first 
and second marks has been printed. The chart 1 may be a 
flat screen of a device such as a notebook computer 
displaying an image of a flat sheet on which first and 
second marks are printed. A screen of glass, such as a 
screen of a liquid crystal display, is suitable for a 
device for displaying the chart 1 since its surface expands 
and contracts with changes in moisture and temperature much 
less than that of a paper sheet. 

[0046] The first marks are used for measurement of 
approximate mark positions and correlating a pair of images 
for calibration and used to determine the angles at which a 
camera 2 photographed the chart 1. At least three first 
marks are provided on the chart 1. Preferably, four first 
marks are provided on the chart 1, one in each quadrant. 
The second mark designates the position of image data of 
the chart 1 photographed with the camera 2 and is also 
referred to as target. Preferably, the second marks are 
provided all over the chart 1 at a uniform density. At 
least thirty, preferably 100 to 200, second marks are 
provided on the chart 1. The chart 1 will be described 
later in detail. 

[0047] The camera 2 is a variable focal length camera, for 
example, as an object of calibration. The camera 2 can be 
functioned as a wide-angle lens camera when the focal 
length is shortened (to 8 mm, for -example) or as a 
telephoto lens camera when its focal length is extended (to 
200 mm, for example) . The focal length of the camera 2 can 
be also set to a value equivalent to that of a standard 
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lens (35 mm) . The camera 2 preferably has an auto-focus 
function, and the lens can be moved to adjust the focus. 
Typically, the camera 2 has a lens with large aberration as 
in the case with an optical camera or a digital camera for 
a general purpose as compared with a camera for 
photogrammetry and photographic measurement. The camera 2 
is used to photograph an object or the chart 1, and, 
typically, used to photograph the object or the chart 1 in 
stereo at a pair of right and left photographing positions 
2R and 2L. Data of a pair of images photographed at the 
right and left photographing positions 2R and 2L are useful 
for three-dimensional image measurement of an object. 
[0048] A calibration data measuring part 100 has an image 
data storing part 110, a focal length data storing part 
120, a calibration element calculating part 130, an image 
processing part 140, a display part 150, and an internal 
parameter function calculating part 160. The image data 
storing part 110 is a recording device for storing image 
data of the chart 1 photographed with the camera 2 and may 
be an electromagnetic recording medium such as a magnetic 
disc or a CD-ROM. The image data to be stored in the image 
data storing part 110 are preferably stereo image data 
photographed with the camera 2 in such a manner that the 
chart 1 can be viewed in stereo. Typically, the data are 
images photographed at the right and left photographing 
positions 2R and 2L. Preferably, images are stored in the 
image data storing part 110 in such a state that the angles 
at which the images were photographed can be determined. 
The calibration data measuring part 100 is provided with an 
I/O device for reading the image data in the image data 
storing part 100. 

[0049] The focal length data storing part 120 stores the 
focal lengths of the camera 2 at which the images stored in 
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the image data storing part 110 were photographed- The 
focal lengths may be focal length information which is 
attached to the image data of the chart 1 photographed with 
the camera 2, focal length data taken out of the camera 2, 
or focal lengths calculated from the image data at the time 
of measurement. When the internal parameters of the camera 
are obtained in the calibration element calculating part 
130, a polynomial for approximating to a function is 
selected in the internal parameter calculating part 160 so 
that they can match the number of measuring points at which 
the focal length data are measured and the intervals 
thereof. For example, when the focal length of the zoom 
camera 2 can be adjusted from 7 . 2 mm to 50 . 8 mm and when 
the number of the measuring points is six, the measurement 
is performed at equal intervals in an optical sense, for 
example, at focal lengths of 7.2 mm, 8.7 mm, 11.6 mm, 24.8 
mm, 39.0 mm and 52.2 mm. The focal length of 7.2 mm to 
50.8 mm of the zoom camera 2 equivalent to 28 mm to 200 mm 
in 35 mm format. 

[0050] The calibration element calculating part 130 has a 
mark extracting part 131, an approximate mark position 
measuring part 132, a precise mark position measuring part 
133, an internal parameter calculating part 134, a mark 
coordinate storing part 135, and a calculated internal 
parameter value storing part 136. The calibration element 
calculating part 130 has an image processing part 140 to 
confirm the internal parameters calculated in the internal 
parameter calculating part 134, and has an image data 
storing part 110 and a display part 150 as external 
devices. As the calibration element calculating part 130 
and the image processing part 140, computers mounting a 
Pentium (trademark) or Celeron (trademark) produced by 
Intel Corporation as a CPU may be used. 
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[0051] The mark extracting part 131 perform a first mark 
extracting process to extract the first marks from the 
image data stored in the image data storing part 110 and 
obtain the image coordinate values of the first marks. The 
first mark extracting process as a pre-process prior to 
calculating and correlating the approximate positions of 
the second marks is performed by The approximate mark 
position measuring part 132. The image coordinate values 
of the first marks are stored in the mark coordinate 
storing part 135. When the first mark includes a mark 
which is the same as the second mark, the positions of the 
second marks in the first marks can be used as the image 
coordinate values of the first marks. The process of 
extracting the first marks by the mark extracting part 131 
will be described late in detail. 

[0052] The approximate mark position measuring part 132 
performs projection conversion to obtain exterior 
orientation element from the image coordinate values of the 
first marks, and calculates the approximate positions of 
the second marks using the single-photograph orientation 
theorem and a collinearity condition expression to 
correlate a pair of images for calibration. ,The process of 
calculating the approximate positions of the second marks 
by the approximate mark position measuring part 132 will be 
described later in detail. 

[0053] The precise mark position measuring part 133 
recognizes the second marks on the paired images for 
calibration and calculates the positions of the second 
marks precisely. The internal parameter calculating part 
134 extracts a second mark suitable for calibration from 
the second marks the precise positions of which has been 
calculated in the precise mark position measuring part 133 
and adjusts the exterior orientation elements and the 
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objective coordinates thereof simultaneously, and 
calculates the internal parameters of the camera 2 as data 
for calibration. The calculated internal parameter value 
storing part 136 stores the internal parameters of the 
camera 2 calculated in the internal parameter calculating 
part 134. When there are second marks whose positions 
calculated by the precise mark position measuring part 133 
are significantly contradicts the positions of other second 
marks in the image data of the chart 1, the internal 
parameter calculating part 134 removes the positions of 
such second marks. The internal parameters of the camera 2 
calculated in the internal parameter calculating part 134 
are preferably stored in the internal parameter calculating 
part 136 together with the focal length of the camera 2 at 
which the chart 1 was photographed. 
[0054] The internal parameters of a camera are the 
principle point position, the screen distance and the 
distortion parameters. Although only distortion parameters 
are herein obtained, the spherical aberration, coma, 
astigmatism and curvature of field comprising Seidel's five 
aberrations may be obtained. The internal parameters 
obtained in the internal parameter calculating part ,134 are 
displayed on the display part in graphical form. The 
process of obtaining the internal parameters of the camera 
by the internal parameter calculating part 134 as well as 
the precise mark measuring part 133 will be described later 
in detail . 

[0055] The image processing part 140 rearranges the data of 
an image photographed with the camera 2 (especially, an 
image of other than the chart 1) using the internal 
parameters calculated in the internal parameter calculating 
part 134. Then, the image photographed with the camera 2 
is displayed on the display part 150 as a low distortion 
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image almost free from lens aberration. The display part 
150 is an image display device such as a CRT or a liquid 
crystal display. The mark coordinate storing part 135 
stores the image coordinate values of the first marks and 
the management numbers and the image coordinate values of 
the second marks. 

[0056] The internal parameter function calculating part 160 
calculates internal parameter functions using the data for 
calibration calculated in the internal parameter 
calculating part 134 and the focal lengths of the camera 2 
at which the images of the chart 1 targeted by the 
calibration element calculating part 130 were photographed. 
The process of calculating data for calibration (internal 
parameter functions) corresponding to the focal lengths of 
the camera 2 by the internal parameter function calculating 
part 160 will be described later in detail. 
[0057] Description will be next made of the chart 1 as a 
chart for calibration. Although description will be herein 
made taking a paper or plastic sheet on which prescribed 
marks are printed, the chart 1 may be a flat screen on 
which an image of the prescribed marks is displayed. 
[0058] FIG. 2 is a plan view, illustrating an example of a 
chart for calibration. The chart 1 is a flat sheet having 
first marks easy to see and a plurality of dots as the 
second marks printed thereon. Five first marks are 
provided on the chart 1. Each of the first marks is a 
rhombus with a mark which is the same as the second mark at 
the center. When the chart 1 is divided into four 
quadrants, each of the quadrants has one first mark. 
Namely, first marks la, lb, lc and Id are located in the 
top left quadrant, top right quadrant, bottom left 
quadrant, and bottom right quadrant, respectively. A first 
mark le is located at the point of origin. For example, 



the first marks la, lb, 1c and Id are located at the same 
distance "d" from the first mark le. Supposing "h" and "1" 
be the vertical distance from the first marks la or lb to 
the first mark le, and the lateral distance from the first 
marks lc or Id to the first mark le, respectively, the 
distance M d" from the first mark le to the first mark la, 
lb, lc, or Id satisfies the following relation: 

d = (h 2 + 1 2 ) 1/2 (1) 
[0059] The first and second marks are printed in the 
desired size or the size of the first and second marks are 
measured in advance. The values expressing the positions 
where the first and second marks are printed are read in 
the mark coordinate storing part 135 of the calibration 
device and used in calculating and correlating the 
approximate position of the second marks in the approximate 
mark position measuring part 132. The chart 1 may be 
stored in the memory of the computer as image data and 
printed before use at the calibration site. When the 
positions of the first and second marks are stored in the 
calibration device in advance and printed in the positions 
on a sheet, the work can be facilitated because no 
measurement is necessary. Alternatively, the coordinate 
positions of the first and second marks on the chart 1 may 
be measured precisely and stored in the mark coordinate 
storing part 135 in advance. 

[0060] The first marks are used in calculating and 
correlating the approximate positions of the second marks 
and also used as targets to determine the photographing 
directions. Also, when the first marks have a mark which 
is same as the second mark in each center, the first marks 
can be used as templates in precise measurement of the 
second marks by the precise mark position measuring part 
133. 
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[0061] FIG. 3 is an explanatory view, illustrating examples 
of the first mark, in which (A) is a mark constituted of a 
rhombic, (B) is a mark constituted of four arrows and (C) 
is a mark constituted of a black rectangle. In the marks 
(A) and (B) , a mark which is the same as the second mark is 
circumscribed by a rhombic and four arrows, respectively so 
as to be easily recognized by the operator. By using such 
marks easy to recognize as the first marks, the first marks 
can be easily extracted and thus can be easily found even 
when one angle is selected as a photographing angle of the 
camera from a wide range of photographing angle. In the 
mark (C) in FIG . 3, the first mark is a black rectangle and 
the mark in the center is inverted in color from the second 
marks. This is also easy to detect. When the tone of the 
mark (C) is inverted, it can be used as a template of a 
second mark in measurement of second marks by the precise 
mark measuring part 133. 

[0062] FIG. 4 is an explanatory view, illustrating examples 
of the second mark, in which (A) is a black circle, (B) is 
a plus sign, (C) is a double circle, (D) is a letter X, (E) 
is a star, (F) is a black square, (G) is a black rectangle, 
and (H) is a black rhombic. Since a multiplicity of the 
second marks are arranged all over the chart 1, various 
types of mark can be employed as the second mark as long as 
its precise position can be easily measured. 
[0063] Description will be made of the procedure for 
photographing the chart 1 with a camera 2 as an object of 
calibration. FIG. 5 is a perspective view illustrating the 
positions of a zoom camera in measuring the lens aberration 
at various focal lengths of the camera. Calibration can be 
performed when there are at least two. images of the chart 1 
photographed from different angles. When a flat chart 
printed on a sheet is used as the chart 1, the chart 1 is 
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preferably photographed from at least three angles. 
Thereby, stable and reliable measurement of calibration 
elements, focal length, in particular, can be performed. 
FIG. 5 shows the procedure for photographing the chart 1 
from the front (I), upper left side (II), upper right side 
(III), lower left side (IV) and lower right side (V). The 
incident angle of the optical axis of the camera 2 with 
respect to the flat chart is preferably in the range of 10 
to 30 degrees when the depth accuracy on site is set to 
about 1 cm. In view of the fact that the distance range in 
which the camera can be focused is limited because of the 
focal depth of the lens, the incident angle is preferably 
in the range of 12 to 20 degrees. Typically, the incident 
angle is 15 degrees. The "various focal lengths'' means the 
focal lengths equivalent to those of normal lens, wide- 
angle lens and telescopic lens in a single lens reflex 
camera. 

[0064] Description will be made of the procedure for 
photographing the chart 1 with a zoom camera with reference 
to FIG. 6. In FIGS. 6(A), 6(B), 6(C) and 6(D) are images 
photographed with the camera and 6(E), 6(F), 6(G) and 6(H) 
show the positional relations between the chart 1 and the 
camera 2 corresponding to the images. The following (I) to 
(V) in correspond to the camera positions in FIG. 5. 
[0065] (I) : An image is photographed such that the image 
area is filled with all the first and second marks (FIG. 
6(A), 6(E)). At this time, the marks in the peripheral 
areas are preferably positioned as close to the edges of 
the image area as possible. Thereby, the distortion in the 
peripheral areas, as well as in the central area, of the 
lens can be reliably corrected. Thus, the object distance 
H is changed according to the focal length of the camera. 
[0066] (II): When the focal length of the zoom camera is 
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equivalent to that of a telephoto lens or a normal lens, 
the camera is moved to a position at a distance about 1/3 
the object distance H from the front position so that the 
first mark la, for example, in the upper left quadrant of 
the chart 1 is located at the center of the image area 
(FIG. 6(B), 6(F)). When the focal length of a zoom camera 
is equivalent to that of a wide-angle lens and the when the 
object .distance H is within 1 m, the camera 2 is moved so 
that the target first mark is located in front of it. 
Then, the camera 2 is turned so that the first mark le at 
the center of the chart 1 comes to the center of the image 
area with its position maintained (FIG. 6(C), 6(G)). The 
camera 2 is then moved closer to the chart 1 so that the 
first and second marks fill the image area, and the image 
is photographed (FIG. 6(D), 6(H)). 

[0067] (III): The camera is moved so that the first mark lb 
in the top right quadrant of the chart 1 comes to the 
center of the image area. Then, the camera is turned so 
that the first mark le at the center of the chart 1 comes 
to the center of the image area. The camera 2 is then 
moved closer to the chart 1 so that the first and second 
marks fill the image area, and the image is photographed. 
[0068] (IV) : The camera is moved so that the first mark lc 
in the bottom left quadrant of the chart 1 comes to the 
center of the image area. Then, the camera is turned so 
that the first mark le at the center of the chart 1 comes 
to the center of the image area. The camera 2 is then 
moved closer to the chart 1 so that the first and second 
marks fill the image area, and the image is photographed. 
[0069] (V) : The camera is moved so that the first mark Id 
in the bottom right quadrant of the chart 1 comes to the 
center of the image area. Then, the camera is turned so 
that the first mark le at the center of the chart 1 comes 
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to the center of the image area. The camera 2 is then 
moved closer to the chart 1 so that the first and second 
marks fill the image area, and the image is photographed. 
[0070] Through the above procedure, the angle of the camera 
2 can be obtained as the difference in the photographing 
angle, the focal lengths can be reliably measured. 
[0071] Description will be made of the advantage of the 
photographing procedure utilizing the first marks at the 
time when a sheet on which the first and second marks are 
printed or a flat screen on which the first and second 
marks are displayed is used as a chart 1. When marks 
printed on a flat sheet are photographed, a camera cannot 
be inclined at a desired angle and, thus, the screen 
distance (focal length) cannot be obtained precisely. 
Namely, since the chart has no change in the focal length 
direction (height direction or depth direction) , there is 
nothing to constrain the calculated values of the internal 
parameters. Thus, when the internal parameters of the 
camera are calculated, the values are not reliable. Thus, 
three-dimensionally arranged targets are measured to obtain 
the focal length. However, three-dimensionally arranged 
targets are difficult to measure and the measurement cannot 
be automated. Also, three-dimensionally arranged targets 
are difficult to produce. 

[0072] The flat chart 1 is, however, a flat sheet on which 
the first and second marks are printed or a screen on which 
the first and second marks are displayed, so that the 
difficulty of correlating the targets in using three- 
dimensionally arranged targets can be solved. Also, when 
photographing is performed using the first marks, the 
camera 2 can be inclined at a desired angle. Thereby, the 
change in the height (depth) direction is produced, and the 
focal length can be calculated precisely. For example, 
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when the camera 2 can be inclined by 10 degrees or more, 
the focal length can be reliably obtained with the 
calibration device of the present invention. 
[0073] The distance H between the camera 2 and the chart 1 
is obtained from the focal length f of a zoom camera. For 
example, when the focal length of a zoom camera is 
equivalent to that of a normal lens of a 35 mm camera, the 
object distance H is about 90 cm. The distance N 'd" between 
the first marks on the chart 1 is 20 cm, for example. 
Thus, when the photographing direction is inclined from the 
front position (I) to the upper left position (II) and so 
on, a photographing angle of about 10 degrees can be 
secured. 

[0074] The upper limit of the inclination angle of the 
photographing direction is determined by the focal depth 
and so on. Namely, when the inclination angle of the 
photographing direction is large, the distances between the 
camera 2 and the first marks vary depending upon the first 
marks and of the first marks in the image are blurred. 
Thus, the upper limit of the inclination angle of the 
photographing direction is 30 degrees, for example. In 
reality, when the chart 1 is photographed such that the 
first and second marks fill the image area according to the 
steps (I) to (V), the above condition is automatically met. 
Thus, the conditions on the object distance and the 
photographing position are satisfied. 

[0075] FIG. 7 is a view illustrating a camera distance in 
measuring the lens aberration when the focal length of a 
zoom camera is equivalent to that of a normal lens or a 
telephoto lens. When the focal length of a zoom camera is 
equivalent to that of a normal lens or a telephoto lens, 
the viewing angle to the photographing lens is narrow and 
the camera cannot be inclined very largely. Thus, when the 
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photographing angle is inclined from the front position (I) 
to the top left position (II) and so on, a photographing 
angle of 10 degrees cannot be secured. This is because the 
distance H between the camera 2 and the chart 1 is 1 m or 
longer and the distance "d" between the first marks is 
about 20 cm when the focal length is long. Thus, the 
camera positions (II) and (IV) on the left side and the 
camera positions (III) and (V) on the right side are 
determined with respect to the front position (I) . At this 
time, the camera is shifted by a distance of about one- 
third the object distance H from the front position (I) . 
Then, photographing at the top left position (II), bottom 
left position (IV), top right position (III) and bottom 
right position (V) are performed. The optical axis of the 
camera is aligned with the normal line of the chart 1 or 
may be directed toward the chart 1. 

[0076] In the above embodiment, description has been made 
of an example in which the chart 1 is photographed from 
five different positions: front (I),- top left (II), top 
right (III), bottom left (IV) and bottom right (V). 
However, it is only necessary that the chart 1 is 
photographed from two different positions: right and left. 
Alternatively, the chart 1 may be photographed from three 
different positions. When the chart 1 is photographed from 
two different angles, a photographing angle of about 10 
degrees must be secured. 

[0077] Description will be made of the flow of the process 
in the device for measuring data for calibration of the 
present invention. FIG. 8 is a flowchart for explaining 
the method for measuring data for calibration using the 
device for measuring data for calibration. Calibration of 
a camera is a preparatory process of three-dimensional 
measurement using the camera and thus is herein referred to 
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as preparatory process. The preparatory process is started 
with setting the chart 1 (S100) . Then, the focal length of 
the camera 2 is set to a target value (S102) . For example, 
when the focal length of the zoom camera 2 can be adjusted 
from 7.2 mm to 50.8 mm and when the focal length data are 
collected at six points, one of the values spaced at equal 
intervals in an optical sense (7.2 mm, 8.7 mm, 11.6 mm, 
24.8 mm, 39.0 mm and 52.2 mm) is selected. 
[0078] Then, chart 1 is photographed with the camera 2 as 
an object of calibration (S104) . The photographing is 
performed according to the procedure described with 
reference to FIG. 5 and FIG. 6 in when the lens aberrations 
at the time when the focal length of the zoom camera is 
equivalent to that of a normal lens or a wide lens is 
measured, and performed according to the procedure 
described with reference to FIG. 7 when the lens aberration 
at the time when the focal length of the zoom camera is 
equivalent to that of a telephoto lens is measured. Then, 
the mark extracting part 131 extracts the marks on the 
chart from the photographed images of the chart 1 (S105) . 
At this time, the approximate mark position measuring part 
132 and the precise mark position measuring part 133 
measure the positions of the marks extracted by the mark 
extracting part 131. Then, the internal parameter 
calculating part 134 calculates the calibration elements of 
the camera 2 (S106) . The operations of the component parts 
of the calibration element calculating part 130 (the mark 
extracting part 131, the approximate mark position 
measuring part 132, the precise mark position measuring 
part 133 and the internal parameter calculating part 134) 
will be described later in detail with reference to FIG. 9 
and FIG. 10. Then, it is judged whether there still 
remains a focal length of the camera 2 at which measurement 
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should be performed (S108) . If there still remains a focal 
length, the process returns to S102. Otherwise, the 
internal parameter function calculating part 160 calculates 
functions of the calibration elements which take the focal 
length as a parameter (S110) . Thereby, the preparatory 
process is completed. 

[0079] FIG . 9 is a flowchart for explaining the process of 
calculating the calibration elements at each focal length. 
The image data photographed at each focal length from the 
various angles with the camera 2 in step S104 has been 
stored in the image data storing part 110. Then, the step 
S105 is started, a subroutine shown in S200 is executed. 
The calibration device reads the image data stored in the 
image data storing part 110 and displays the data on the 
display part 150 (S202) . Then, the operator selects images 
on which correlating and measurement of targets are 
performed from the images displayed on the display part 150 
(S204). Then, the mark extracting part 131 extracts the 
first marks from the selected images (S206) . 
[0080] (I): First Mark Extraction Process 
[0081] In the first mark extraction process, in order to 
determine second-order equations for projection conversion 
of the plane coordinates of the chart 1 into image 
coordinates (camera side) , the positions of at least three 
first marks out of the first marks on the plain coordinate 
system are measured on the image data. Here, since the 
first marks include the second marks therein, the positions 
of the first marks can be designated precisely by 
designating the positions of the second marks included in 
the first marks. In the first mark extraction process, the 
steps I-(l) to I- (4) are repeated for all the first marks. 
For example, in the case of the chart 1 shown in FIG. 2, 
the process is performed on the first marks la, lb, lc and 
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Id. 

[0082] I-(l): The operator points the cursor of the mouse 
to the second mark in the first mark to be detected on the 
entire image displayed on the display part 150 and clicks 
the mouse thereon to obtain the approximate position of the 
first mark. 

[0083] I- (2): The operator designates an area including 
the coordinates of the first mark obtained in step I-(l) 
and the second marks around it from an enlarged view and 
displays it. At this time, the image including the second 
marks can be used as a template in measuring the precise 
positions of the second marks. 

[0084] I-(3): The operator points the cursor to the 
gravity center of the second mark on the enlarged image 
displayed in the step I- (2) and clicks the mouse thereon to 
make the coordinates the position of the gravity center of 
the first mark. The positioning in step I- (3) may not be 
precise because correlating the approximate positions will 
be performed in a later process. 

[0085] I-(4): The operator inputs the management number of 
the second mark corresponding to the position of the 
gravity center of the first mark measured in step- 1- (3) to 
correlate it to the management number of the second mark 
stored in the mark coordinate storing part 135. At this 
time, the position of the gravity center of the first mark 
measured in step I- (3) is attached to the management number 
of the second mark as reference coordinates. 

[0086] In the first mark extraction process, when the order 
of measuring the first marks, for example, on the chart 1 
are determined in advance, the number attaching process can 
be automatically performed on the mark extracting part 130 
side even if the operator does not input the management 
number of the second mark. In the first marks extraction 
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process, for example, the screen of the display part 150 
may be divided into two sections and the entire image as 
shown in FIG. 2 and the enlarged image as shown in FIG. 3 
(A) or FIG. 3(B) may be displayed in each section to 
facilitate the operator's work. 

[0087] The first marks extraction process may be performed 
using only an entire image as shown in FIG. 2 without using 
an enlarged image. In this case, the step I-(l) is 
performed and then the process performed in step I- (4) in 
the above process, namely the input of the management 
number of the second mark corresponding to the position of 
the gravity center of the first mark measured in step I-(l) 
is performed. Thereby, the steps I- (2) and I- (3) can be 
omitted because no enlarged image is used. However, since 
the entire image of the chart is displayed, the first marks 
are displayed in small sizes. The operator may decide 
whether to use an enlarged image or not on its preference. 
[0088] Description will be made of the case where the first 
mark extraction process is automatically performed by the 
mark extracting part 131. At first, the external portion 
of the first mark excluding the second mark is registered 
as a template. The external portion of the first mark 
processed first in the first mark extraction process is 
registered as a template image. Then, the other first 
marks can be automatically measured by template matching. 
Also, the correlating of the first marks can be performed 
with ease since the positions of the first marks are 
apparent on the image. For example, when the first marks 
are arranged as shown in FIG. 2, it is easy to correlate 
the five first marks based on the detected coordinates 
thereof. The template matching is the same as the target 
recognizing process (S302) in measurement of precise 
positions of the second marks and thus is not described 
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here . 

[0089] Description will be made of the case where the first 
mark extraction process is performed more automatically by 
the mark extracting part 131- A template image of the 
first mark for use in the first mark extraction process is 
registered in the mark extracting part 131 in advance. 
Then, the first marks are individually extracted by 
template matching using the template image of the first 
mark. Thus, the work of designating the first mark in step 
I-(l) can be omitted. Namely, when the first marks are 
apparently different from the second marks, the process can 
be automated by registering a temporal template image in 
the mark extracting part 131. 

[0090] However, since at least three first marks must be 
measured, the process can be easily performed manually. 
[0091] Description will be made with reference again to 
FIG. 9. The approximate mark position calculating part 132 
measures and correlates the positions of the second marks 
(S208) This process include a step of obtaining exterior 
orientation elements (II-l) and step of calculating the 
approximate positions of the second marks (II-2). 

[0092] (II-l) : Step of obtaining exterior orientation 
elements 

[0093] The approximate mark position measuring part 132 
substitutes the image coordinates of the first marks 
obtained in step S206 and the reference coordinates 
corresponding thereto into the second-order equations (2) 
for projection conversion to set up an observation equation 
for obtaining parameters bl to b8: 

X = (bl-x + b2-y + b3)/(b7-x + b8-y + 1) 
Y - (b4-x + b5-y + b6)/(b7-x + b8 • y + 1) (2) 
wherein X and Y represent the reference point coordinates 
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and x and y represents the image coordinates. 
[0094] The relation between the reference point coordinates 
and the image coordinates will be described. FIG. 11(A) is 
an explanatory view of an image coordinate system and an 
objective coordinate system in center projection. In 
center projection, a objective coordinate system 52 as a 
reference point coordinate system on which the chart 1 is 
located and an image coordinate system 50 on which the film 
in the camera 2 or a CCD is located are in the positional 
relation as shown in FIG . 11(A) with respect to the project 
center Oc. Here, (X, Y, Z) is the coordinates of an object 
such as a reference mark on the objective coordinate system 
52, (X0, Y0, Z0) are the coordinates of the projection 
center Oc, (x, y) are the coordinates of a point on the 
image coordinate system 50, C is the screen distance from 
the projection center Oc to the image coordinate system 50, 
and go , cp and k are the inclinations of the image coordinate 
system 50 in photographing with respect to the three axes 
X, Y and Z, respectively, forming the objective coordinate 
system 52 and referred to as exterior orientation element. 
[0095] Then, using the parameters bl'to b8 in the equation 
(2) , the following exterior orientation elements are 
obtained by the equations (3) : 
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(3) 

Wherein, Al - 1 + tan 2 cp, A2 = Bl + B2 ■ tancp/sinco, A3 = B4 + 
B5 • tancp/sino, A4 =' tancp/ (coscp • tanco) , Zm is the average of 
the heights of the reference points la, lb, lc and Id, and 
C is the focal length and corresponds to the screen 
distance. Here, the reference points la, lb, lc and Id are 
on a plain coordinate system and thus assumed to form a 
uniform height plane. 

[0096] (II-2) : Step of calculating approximate position of 
second marks 

[0097] Camera coordinate (xp, yp, zp) on an inclined camera 
coordinate system represented by the image coordinate 
system 50 corresponding to the coordinates (X, Y, Z) of an 
object on the ground represented by the objective 
coordinate system 52 are given by the equation (4) base on 
the principle of single photograph orientation: 
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wherein, (X0, Y0, Z0) are the ground coordinates of the 
projection center Oc as shown in FIG. 11(A). 
[0098] Then, the inclinations (go, q>, k) obtained using the 
equations (3) are substituted into the equation (4) and a 
rotation matrix calculation is performed to obtain rotation 
matrix elements all to a33. 

[0099] The thus obtained rotation matrix elements all to 
a33, the coordinates of the position of the camera (X0, Y0, 
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ZO) obtained using the equations (3) and the reference 
point coordinates (X, Y, Z) of a target are substituted 
into the equation of collinearity condition (5) to obtain 
he image coordinates (x, y) of the target. The equation of 
collinearity condition is a relationship equation which 
holds when the projection center, a photographed image and 
an object on the ground are on one line. Thereby , the 
positions of the second marks without lens aberration are 
calculated. Thus, the approximate image coordinates of the 
target in an image photographed with the real camera 2 with 
lens aberration can be obtained: 

x = -O {all (X-XO) + al2(X-X0) + al3(Z-Z0)}/ 
{a31(X-X0) + a32(X-X0) + a33(Z-Z0)} 

y = -C- {a21 (X-XO) + a22(X-X0) + a23(Z-Z0)}/ 

{a31(X-X0) + a32(X-X0) + a33(Z-Z0)} (5) 
[0100] In the calculation of tan in the equations (3), 
two solutions are obtained. Thus, each of the inclinations 
co, cp and k has two solutions. Here, all the solutions are 
calculated and correct go, cp and k are calculated by 
comparing the residuals between the image coordinates of 
the four first marks la, lb, lc and Id and the image 
coordinates of the corresponding points obtained using the 
equations (5) . 

[0101] Although second order equations for projection 
conversion are used, the present invention is not limited 
thereto. Other equations for projection conversion such as 
third equations for projection may be used. 
[0102] The approximate mark position measuring part 132 
correlates the second marks by assigning the management 
numbers of the second marks attached to a reference point 
file stored in the mark coordinate storing part 135 or the 
like to the target (second mark) of each of the first 
marks. 
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[0103] Description will be made with reference again to 
FIG. 9. The precise mark position measuring part 133 
measures the precise positions of the second marks (S210) . 
The procedure for measuring the precise positions of the 
second marks will be described in detail with reference to 
FIG. 10. The precise mark position measuring part 133 
recognized the targets as second marks (S302) . Template 
matching using normalized correlation, for example, is used 
for the target recognition. The targets recognition will 
be described in detail. 
[0104] (III) Target recognition 

[0105] FIG. 11(B) is an explanatory view of a template 
image for normalized correlation and an object image for 
use in the target recognition. At first, an arbitrary 
target is selected from the gravity centers of the targets 
of the first marks, such as the first marks la, lb, lc and 
Id measured in the first mark extracting step (S206) . The 
template image for normalized correlation is an M * M pixel 
image centered around the gravity center (image 
coordinates) of the target. The object image is an N x n 
pixel image centered around the approximate position (image 
coordinates) of the target calculated in the step of 
measuring the approximate positions of the second marks 
(S208) . 

[0106] Then, template matching according to the normalized 
correlation expressed by the equation (6) is performed on 
the object image and the point where the correlation value 
is maximum is obtained. Superposition is accomplished at 
the point where the correlation value is maximum and the 
target is regarded as being reqognized at the point. The 
coordinates of the center of the template image are 
converted into image coordinates on a same-scale image, 
which are determined as a detecting point: 
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A = {M 2 x S(Xi x Ti) -EXi x STi}/ 

[{M 2 x EXi 2 -E(Xi) 2 } x{M 2 x ETi 2 -E(Ti) 2 }] 1/2 

(6) 

wherein, A represents the correlation value, M represents 
the side size of the template image, Xi represents the 
object image and Ti represents the template image. The 
side sizes N and M are variable but preferably as small as 
possible to shorten the processing time based on the 
premise that the target can be sufficiently included in the 
images . 

[0107] Description will be made with reference again to 
FIG. 10. Sub-pixel edge detection is performed on the 
second marks (S304). The object image on which the sub- 
pixel edge detection of the second marks is performed is an 
N x n pixel image centered around the detecting point 
recognized as a target in step S62. Laplacian-Gaussian 
filter (LOG filter) as a quadratic differential of a Gauss 
function expressed by the equation (7) is applied to the 
brightness waveform in the object image and the two zero 
crossing points on a curve as a result of calculation, 
namely the edges, will be detected with sub-pixel accuracy: 

V 2 -G(x) = { (x 2 - 2o 2 ) /2na 6 } • exp (-x 2 /2a 2 ) (7) 
wherein, o represents the parameter of the Gauss function. 
To detect with sub-pixel accuracy means to perform position 
detection with higher accuracy than one pixel. 
[0108] Then, the gravity center of the target is detected 
(S306) , and the process is returned (S308). Here, the 
position where of the cross point of edges in the x and y 
directions cross each other obtained using the equation (7) 
is determined as the position of the gravity center of the 
target. The measurement of the precise positions of the 
second marks is not necessarily performed by the process 
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comprising steps S302 to S306. The precise positions of 
the second marks may be obtained by another gravity center 
position detection method such as a moment method or a 
modified template matching method. 

[0109] Description will be made with reference again to 
FIG. 9. It is confirmed that there is no apparent error in 
the positions of the gravity centers of all the targets 
(S212) . Namely, it is judged whether the position 
detection of the recognized targets was appropriate. For 
the convenience of the operator, the positions of the 
detected targets are displayed on the display part 150. 
When there is no error, the process goes to step S216. 
When there is error, inappropriate target positions are 
corrected (S214). Targets the correlation values of which 
calculated in step S302 are low or targets the detected 
gravity center positions of which are far apart from their 
approximately positions are displayed on the display part 
150 in such a manner that the operator can easily 
recognize, in red, for example. The operator manually 
recalculates the positions of such targets (designates the 
gravity center positions thereof with the mouse) . The 
erroneous target positions are not necessarily corrected 
here. They can be removed since they are detected as 
abnormal points in the process for obtaining calibration 
parameters in step S218. 

[0110] The process comprising steps S204 to S214 are 
performed on for all the images necessary for the 
measurement of the lens aberration (S216) . For example, 
when five images were photographed, the process may be 
performed on the five images. When sufficient number of 
images for the measurement of the lens aberration has been 
processed, the other photographed images may not be 
necessarily processed. 
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[0111] When a sufficient number of images for the 
measurement of the lens aberration have been processed, a 
process for obtaining calibration elements for the lens 
aberration using the process for calculating the internal 
parameter performed in the internal parameter calculating 
part 134 is performed (S218) . The calculation of 
calibration elements is performed on all the second marks 
which have been correlated and whose gravity centers have 
been obtained by the processes in the approximate mark 
position measuring part 132 and the precise mark position 
measuring part 133. 

[0112] (IV) : Process for calculating internal parameters of 
camera (bundle adjustment with self-calibration) 
[0113] For the process for calculating the internal 
parameters of the camera in the internal parameter 
calculating part 134 is, ''bundle adjustment with self- 
calibration" used in the field of photogrammetry is used. 
The "bundle adjustment" is a method in which an observation 
equation is set up for each of light bundles of each image 
based on the collinearity condition that light bundles 
connecting the object, lens and CCD surface should be on 
one line, and the position and the inclination of the 
camera (exterior orientation elements) and the coordinate 
positions of the second marks are simultaneously adjusted 
by a least square method. With the "bundle adjustment with 
self -calibration" , the calibration elements, namely the 
inner orientations of the camera (lens aberration, 
principle point position and focal length) can be also 
obtained. The collinearity condition basic equations of 
bundle adjustment with self -calibration (which will be 
hereinafter referred to as "bundle adjustment") are the 
following equations (8) and (9) : 
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an(X-Xo)+ a 12 (Y-Yo)+ a 13 (Z-Zo) 
° a 31 (X-Xo)+ a 32 (Y-Yo)+ a 33 (Z-Zo) 



+ A x 



(8) 



si 

U 



a 21 (X-Xo)+ a 22 (Y-Yo)+ a 23 (Z-Zo) 
° a 31 (X-Xo) + a 32 (Y-Yo)+a 33 (Z-Zo) 



+ A y 



(9) 



[0114] The equations (8) and (9) are based on the equation 
of collinearity condition (5) for single-photograph 
orientation described in describing the first mark 
extraction process. Namely, the bundle adjustment is a 
method in which various solutions are obtained from a 
plurality of images by least square approximation, and by 
which the exterior orientation positions of the camera at 
different photographing positions can be obtained at the 
same time. Namely, the calibration elements of the camera 
can be obtained. 

[0115] The following equation (10) is an example of a 
correction model for an inner orientation (lens aberration) 
applied to a lens with distortion in the radial direction: 



r A x = x 0 + x (k ir 2 + k 2 r 4 ) 
< Ay-y 0 + y (k ir 2 +k 2 r 4 ) 
, r 2 =(x 2 +y 2 )/c 2 



(1 0) 



kj , k 2 : Lens distortion in the radial direction 
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[0116] The correction model is not limited to the above 
one. A correction model suitable for the lens may be 
selected. The calculation can be made by a successive 
approximation method when there are at least 6 reference 
points on the ground coordinate system and the image 
coordinate system. The internal parameter calculating part 
134 has a threshold value of the successive approximation 
method and removes the second marks on the chart 1 whose 
error is not smaller than the threshold value to obtain 
precise calibration elements. Thus, the second marks which 
were not detected as erroneous marks in the step of 
confirming the gravity centers of the targets (S212) can be 
detected and removed in step S218. 

[0117] Description will be made with reference again to 
FIG. 9. The result of calculation for obtaining the 
calibration elements by the internal parameter calculating 
part 134 is judged (S220) . When the calculation did not 
converge or there are inappropriate calibration elements in 
the obtained calibration elements, the problems are solved 
in step S222. In step S222, images including erroneous 
second marks are selected. Since it has become apparent 
which second marks of which images have error by the 
internal parameter calculating part 134 when the 
calibration in step S218 was completed, the detecting 
points of the targets are displayed for confirmation. 
[0118] The operator manually corrects the erroneous second 
marks (S224). Namely, since the coordinates of the gravity 
center of the erroneous second marks are shifted, the 
correction is made by moving the mark displayed as 
erroneous second marks to the gravity center position 
displayed as being appropriate. Then, it is judged whether 
the correction of the positions of the erroneous second 
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marks has been completed (S226) . When the correction has 
been completed, the process returns to the step of 
calculating the calibration elements in step S218, and the 
calibration elements are calculated again. If there are 
other second marks to be corrected, the process returns to 
step S222 and the operation for correcting the positions of 
erroneous second marks is repeated. 

[0119] When the result of operation for obtaining the 
calibration elements is appropriate, the result is 
displayed on the display part 150 (S228) . FIG . 12 is an 
explanatory view illustrating an example of a screen 
displaying the result of the operation for obtaining the 
calibration elements. For example, the focal length, 
principle point position and distortion parameters as the 
calibration elements are displayed on the display part 150. 
As for distortion representing the lens aberration, curves 
142 before correction, curves 144 after correction, and 
ideally corrected curves 146 may be displayed in graphical 
form for easy understanding. 

[0120] In addition, an image whose distortion has been 
corrected based on the result of the calibration may be 
formed in the image processing part 140 and displayed on 
the display part 150. Thereby, an image displaying device 
which can correct an image photographed with a camera with 
large distortion can be provided. 

[0121] Description will be made of examples of the result 
of the calculation of a function of a calibration element 
which takes the focal length as a parameter performed in 
the internal parameter function calculating part 160 using 
the internal parameters calculated in the calibration 
element calculating part 13.0. FIG. 13 is a view 
illustrating the relation between the focal length and the 
coefficients for use in the internal parameter functions. 
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FIG. 13(A) shows the relation between the focal length and 
a coefficient kl in the equation (10), FIG. 13(B) shows the 
relation between the focal length and a coefficient k2 in 
the equation (10), FIG. 13(C) shews the relation between 
the focal length and a coefficient xq of deviation in the 
x-axis direction of the principle point of the camera from 
the center of the image on an image coordinate system x, 
and FIG. 13(D) shows the relation between the focal length 
and a coefficient y 0 of deviation in the y-axis direction 
of the principle point of the camera from the center of the 
image on an image coordinate system y. As described 
before, the zoom camera 2 in which the focal length can be 
adjusted from 7.2 mm to 50.8 mm is taken here as an 
example. When the focal length data are measured at six 
points, the measurement is performed at focal lengths of 
7.2 mm (wide end) and 52.2 mm (telephoto end) , and 8.7 mm, 
11.6 mm, 24.8 mm, 39.0 mm are selected as the measuring 
point between them to perform measurement at equal 
intervals in an optical sense. 

[0122] The coefficient kl in the equation (10) is maximum 
at the wide end and small on the telephoto side. The 
coefficient X0 varies in a complex manner; it takes the 
minimum value of 4.46 when the focal length of the zoom 
camera 2 is 8.7 mm and 52.2 mm and the maximum value of 
4.55 when the focal length of the camera 2 is 39.0 mm and 
thus is approximated with a curve of the fifth order. The 
coefficient yo varies monotonously with the focal length of 
the zoom camera 2. The chart is photographed in focus, so 
that the focal length f of the zoom camera 2 and the 
distance C from the projection center point Oc to the image 
coordinate system 50 are equal. 

[0123] In the internal parameter function calculating part 
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160, when the focal length f is input, the coefficients kl, 
k2, x 0 and yo for use in the internal parameter functions 
are obtained. Then, the coefficients are substituted into 
the equations (8), (9) and (10) as the internal parameter 
functions to set up observation equations for each of the 
measuring points. Then, by solving the equations 
simultaneously and applying a least square method, the most 
provable internal parameters can be calculated. 
[0124] FIG. 14 is a block diagram illustrating a device for 
measuring data for calibration according a second 
embodiment of the present invention. Although the flat 
chart 1 is used as the chart for calibration in the 
embodiment shown in FIG. 1, a three-dimensional reference 
chart 20 is used in the second embodiment. In FIG. 14, 
those parts corresponding to the components in FIG. 1 are 
identified with the same numerals, and their description 
will be omitted. 

[0125] The three-dimensional reference chart 20 shown in 
FIG. 14 has targets 20a to 20h whose positions have been 
three-dimensionally measured with .precision . The number, 
heights, and plane coordinates are appropriately determined 
so that the chart 20 can be suitable for three-dimensional 
measurement. The camera 2 is used to photograph the three- 
dimensional reference chart 20 in stereo at a pair of right 
and left photographing positions 2L and 2R. The data of a 
pair of stereo images photographed with the camera 2 are 
sent to an image data storing part 110 via an image 
information recording medium or the like. The distance 
between the right and left photographing positions 2R and 
2L, which is referred to as baseline distance, is measured 
precisely . 

[0126] A calibration element calculating part 170 
calculates the internal parameters of the camera 2 when the 
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three-dimensional reference chart is used and has a mark 
extracting part 171, internal parameter calculating part 
173, three-dimensional chart target recording part 175 and 
a calculated internal parameter value recording part 177. 
[0127] The mark extracting part 171 extracts targets 20a to 
20h included in the stereo images of the three-dimensional 
chart 20 and measures the positions of the targets 20a to 
20h on an image coordinate system. The internal parameter 
calculating part 173 calculates the internal parameters of 
the camera 2 as data for calibration using the data on the 
positions of the targets 20a to 20h measured in the mark 
extracting part 171 and the positions of the targets 20a to 
20h stored in a three-dimensional reference chart target 
storing part 175, and adjusts the exterior orientation 
elements and the coordinates of objective points of the 
data of the paired right and left stereo images 
simultaneously. The positions of all the targets 20a to 
20h of the three-dimensional chart 20 are stored in the 
three-dimensional reference chart target storing part 175. 
The internal parameters calculated in the internal ; 
parameter calculating part 173 are stored in the calculated 
internal parameter value storing part 177. 

[0128] The mark extracting part 171 is preferably provided 
with a function of removing the stereo images of the three- 
dimensional reference chart 20 in which the targets 20a to 
20h does not clearly appear. The internal parameters of 
the camera calculated in the internal parameter calculating 
part 173 are preferably stored in the calculated internal 
parameter value storing part 177 together with the focal 
lengths at which the three-dimensional reference chart was 
photographed. 

[0129] Description will be made of the method for measuring 
data for calibration using the device constituted as above 
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with reference again to FIG. 8. First, the three- 
dimensional reference chart 20 is set (S100) , and the focal 
length of the camera 2 is set to a target value (S102) . 
Then, the three dimensional reference chart 20 is 
photographed with the camera 2 as an object of calibration 

(5104) . In this case, the three-dimensional reference 
chart 20 is photographed at a prescribed camera distance 
without adjusting the focal length of the zoom camera. 

[0130] Then, the mark extracting part 171 extracts marks 
formed on the chart 1 from the images taken from the chart 

(5105) . The internal parameter calculating part 173 
calculates the calibration elements of the camera 2 (S106) . 
The operation of the component parts of the calibration 
element calculating part 170 (mark extracting part 171 and 
the internal parameter calculating part 173) is as follows. 
The mark extracting part 171 extracts the targets 20a to 
20h photographed in the stereo images of the three- 
dimensional reference chart 20 and measured the positions 
of the targets 20a to 20h on an image coordinate system. 
Then, the internal parameter calculating part 173 
calculates the internal parameters of the camera 2 as data 
for calibration using the data on the positions of the 
targets 20a to 20h measured in the mark extracting part 171 
and the positions of the targets 20a to 20h stored in a 
three-dimensional reference chart target storing part 175. 
The internal parameters of the camera calculated in the 
internal parameter calculating part 173 are stored in the 
calculated internal parameter value storing part 177 
together with the focal lengths at which the three- 
dimensional reference chart was photographed. 

[0131] Then, it is judged whether there still remains a 
focal length of the zoom camera 2 at which (S108) 
measurement should be performed. If there still remains a 
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focal length, the process returns to S102 and a calibration 
data measuring device 100 measures the internal parameters 
of the camera at the focal length, Otherwise, the internal 
parameter function calculating part 160 calculates the 
functions of the calibration elements which take the focal 
length as a parameter (S110) . Thereby, the preparatory 
process is completed. 

[0132] FIG - 15 is a block diagram illustrating an image 
data processing device according to an embodiment of the 
present invention. Designated as 200 is an image 
processing device. In the image processing device 200, the 
calibration data measuring device 100 performs image 
measurement using stereo images of an object 30 
photographed with the camera 2 whose internal parameters as 
data for calibration have been measured. The object 30 is 
an object having a three-dimensional the three-dimensional 
such as an object in an archeological site, a civil 
engineering site, disaster-prevention work site or an 
urban-planning site or a large-scale structure such as a 
turbine in a power plant, aircraft or a ship. 
[0133] The stereo image data of the object 30 photographed 
with the camera 2 is stored in an electromagnetic recording 
medium such as a flexible disc as a pair of right and left 
stereo images. An image data receiving part 210 is an I/O 
device for receiving the stereo image data of the object 30 
photographed with the camera 2. A device for reading an 
electromagnetic recording medium is used as the image data 
receiving part 210. When the focal length at which the 
stereo images of the object were photographed was measured, 
a focal length data storing part 220 transmits the focal 
length to the image data processing device 200. 
[0134] A calibration data forming part 230 forms internal 
parameters necessary for measurement from the stereo image 
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data stored in the image data receiving part 210, and has a 
focal length calculating part 232, a bundle adjustment 
performing part 234, an internal parameter calculating part 
236, and an internal parameter function calculating . part 
238. The focal length calculating part 232 calculates the 
focal length of the camera 2 at the time of photographing 
using the stereo image data stored in the image data 
receiving part 210 using the bundle adjustment performing 
part 234. When the focal length the camera 2 at the time 
of photographing stored in the focal length storing part 
220 is precise, the values stored in the focal length 
storing part 220 may be used as the focal length calculated 
in the focal length calculating part 232 or may be used as 
the initial values for the calculation in the focal length 
calculating part 232. 

[0135] The internal parameter calculating part 236 
calculates coefficients kl, k2, x 0 and y 0 for use in 
calculation of internal parameter functions in the internal 
parameter function calculating part 238 using the focal 
length calculated in the focal length calculating part 232. 
The internal parameter calculating part 238 has the same 
function as the internal parameter function calculating 
part 160 of the calibration data measuring device 100 and 
calculates the internal parameters of the camera 2. 
[0136] An image processing part 240 has a stereo image 
producing part 242, an image measuring part 244, and an 
orthoscopic image producing part 246. The stereo image 
producing part 242 has a function of performing orientation 
(relative orientation and absolute orientation) and 
displacement correction of the stereo images photographed 
with the camera to form displacement-corrected images and 
is typically constituted of software executable by a 
computer. The displacement-corrected images herein are 
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stereo vision images obtained by correcting the 
displacement in the paired right and left stereo images 
photographed with the camera 2. The image measuring part 
244 performs absolute orientation on the stereo images 
produced in the stereo image producing part 242 to 
calculate the ground coordinates of each pixel. The 
process is disclosed in detail in JP-A-H11-3518 65 filed by 
the same applicant. By using the image measuring part 244 , 
it is possible to measure a three-dimensional configuration 
such as the evenness on a surface of the object 30 
precisely. 

[0137] The orthoscopic image producing part 246 performs 
three-dimensional measurement on the stereo images stored 
in the image data receiving part 210 using the image 
measuring part 244 with the internal parameters of the 
camera 2 calculated in the calibration data forming part 
230 to convert the stereo images into orthoscopic images. 
An orthoscopic image is an orthogonal projection image 
obtained by correcting displacement in an image due to the 
inclination or the specific height of the camera based on 
photogrammetry techniques. The stereo image data stored in 
the image data receiving part 210 are central projected 
images. The orthoscopic image producing part 24 6 converts 
the stereo image data from central projection images into 
orthogonal projection images to form a detailed image 
drawing of the object 30. 

[0138] Description will be made of the relative orientation 
in detail. Orientation calculation is used in the fields 
of aerial photogrammetry and so on, and the positions of 
the right and left photographing devices and so on can be 
obtained through relative orientation and absolute 
orientation. In relative orientation, stereo photographing 
parameters is obtained by the following coplanar 
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conditional equation . 

[0139] FIG . 16 is a view for explaining the orientation 
calculation using a model coordinate system XYZ and right 
and left camera coordinate systems xyz. The origin 0 of 
the model coordinate system is placed at the left 
projection center and a line connecting the origin 0 and 
the right projection center is designated as the X-axis. 
The baseline length 1 is used as the unit length. The 
parameters to be obtained are the following five rotational 
angles: the rotational angle k1 of the Z-axis and the 
rotational angle cpl of the Y-axis of the left camera, and 
the rotational angle k2 of the Z-axis, the rotational angle 
cp2 of the Y-axis and the rotational angle q2 of the X-axis 
of the right camera. The rotational angle go 1 of the X- 
axis of the left camera is zero and thus is not needed to 
be taken into account: 



Xoi Y 01 Z 0 i 1 

Xo2 Y02 Z 0 2 1 

X, Y, Z, 1 

X2 Y2 Z2 1 



= c 



(1 1) 



X 01 ,Y 0 , ,Zoi . a left picture 



Projection center coordinates of 



Xq2 > Yq2 ^ Z 02 



. Projection center coordinates of 
* a right picture 



Xi > Yi % Zi : Image coordinates of a left picture 
X 2 > Y 2 ^ Z 2 • Image coordinates of a right picture 



[0140] Under the above conditions,,, the coplanar conditional 
eguation (12) of the eguation (11) is given as follows and 
the parameters can be obtained by solving the eguation: 
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Y, Z, 

Y 2 Z 2 



Y, Z 2 



Y 2 Z, = o 



(1 2) 



[0141] The following relationship equations hold between 
the model coordinate system XYZ and the camera coordinates 
xyz : 
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[0142] Using the equations (11) to (13), unknown 
parameters are obtained according to the following 
procedure . 

[0143] V-l: The initial approximate value is generally set 
to 0. 

[0144] V-2: The Taylor development is performed on the 
coplanar conditional equation (12) around the approximate 
and a differential coefficient of linearization is obtained 
with the two equations (13) to set up an observation 
equation. 

[0145] V-3: A least square method is applied to obtain a 
correction value with respect to the approximate value. 
[0146] V-4: The approximate value is corrected. 
[0147] V-5: The steps V-2 to V-5 are repeated using the 
corrected approximate value until convergence. 
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[0148] In the above relative orientation calculation, the 
calibration elements calculated in the internal parameter 
function calculating part 238 are used. More specifically, 
the principle point position Ax and the lens aberration AY 
are corrected with the calibration elements. The screen 
distance (focal length) c is calculated using the 
calibration elements . 

[0149] Description will be made of the procedure of the 
three-dimensional measurement using the image data 
processing device constituted as above. FIG. 17 is a 
flowchart for explaining the procedure of photographing an 
object in stereo on site and performing stereo image 
measurement. In onsite processing, an operator goes to the 
site where the object is located and a camera whose 
internal parameter functions have been obtained with a 
calibration data measuring device is used (S400) . The 
operator photographs the object at an arbitrary focal 
length with the camera 2 in an image calculating processing 
manner (S402) . To photograph in an image measurement 
processing manner means to photograph images of an object 
continuously at a distance for stereoscopic viewing as in 
aerial photogrammetry . For example, the images are 
photographed in such a manner that the images are 
overlapped by about 60% in the lateral direction of the 
object and about 20% in the vertical direction of the 
object. The photographed images are stored in an 
electromagnetic recording medium in such a manner as to be 
able to be sent to the image data receiving part 210. 
[0150] The image data stored in. the electromagnetic 
recording medium are received by the image data receiving 
part 210, and a pair of right and left images for producing 
stereo images are selected (S404). The focal length 
calculating part 232 calculates the focal length at which 
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the images were photographed by bundle adjustment with 
self-calibration (S406) . Then,- the internal parameter 
calculating part 236 calculates the other internal 
parameters of the camera 2 corresponding to the calculated 
focal length (S408) . 

[0151] Since the positions and inclination of the stereo 
images are obtained by exterior orientation element 
calculation by relative orientation, the stereo image 
producing part 242 produces final stereo images by adding 
the internal parameters of the camera 2. Then, the image 
measuring part 244 performs three-dimensional measurement 
of the object in the stereo images (S410) . The orthoscopic 
image producing part 24 6 forms orthoscopic images free from 
the effect of the aberration of the lens using the 
calculated parameters of the camera 2 based on the three- 
dimensional measurement data (S412) . Then, it is judged 
whether there still remain paired right and left images for 
producing stereo images in the image data storing part 210 

(S414) . If there remain images, the process returns to 
step S404. Otherwise, the three-dimensional measurement of 
the object can be performed in a region in which the object 
can be viewed stereoscopically . 

[0152] Description will be made of an example in which 
stereo image' measurement is performed in a real field using 
the image data processing device 200. FIG. 18 shows a 
stone wall as an example of the field for stereo image 
measurement. A stone wall has a three-dimensional 
configuration similar to a real field such as a historical 
site or a civil engineering site and thus is suitable for a 
field for experiment. In the field for stereo image 
measurement shown in FIG. 18, there are 49 control points 
represented by the white dots, and their positions have 
been measured with a depth accuracy of ± 1mm. In the image 



data processing device 200, eight points selected from the 
49 control points are used for calculation of the focal 
length of the camera as control points for use in the 
bundle adjustment with self -calibration . The other 41 
control points are used for measurement of depth accuracy 
by stereo image measurement. 

[0153] FIG. 19 is a view illustrating the photographing 
conditions of the camera, and shows the object distance H 
and the photographing baseline length B corresponding to 
the focal lengths equivalent to those of wide, intermediate 
and telephoto lenses. The focal length of the camera 2 
(approximate value) is set to 9 mm in case 1 (wide) , 30 mm 
in case 2 (normal) , and 42 mm in case 3 (telephoto) . The 
size of the photographed area of the stone wall is 2 m * 2 
m. 

[0154] FIG. 20 is a view showing the results of experiment 
for measuring the accuracy of the stereo image measurement, 
showing the focal length (analysis value) [mm], [jam], plain 
accuracy [mm] , depth accuracy [mm] , one-pixel resolution 
a xy on the plane [mm] , and one-pixel resolution o z , in the 
depth direction. The one-pixel resolution o xy on the 
plane, and one-pixel resolution o z , in the depth direction 
are expressed by the following equations, respectively; 

a xy = [H/f] * Qp (14) 
a z = [H/f] * [H/B] x o p (15) 
wherein, H, B, f and o p represent the object distance, the 
baseline length, the focal length and the pixel size. 
[0155] In each column of the FIG. 20, the value in the case 
where the internal parameters of the camera are corrected 
and the value in the case where the internal parameters of 
the camera 2 are not corrected are in the upper raw and the 
lower row, respectively. The depth accuracy is within 2 mm 



50 



at any focal length. On the contrary, when the internal 
parameters of the camera 2 are not corrected, the depth 
accuracy is on the order of 2 cm. Namely, when the 
internal parameters of the camera 2 are corrected according 
to the focal length, the depth accuracy improves by an 
order of magnitude as compared with the case where the 
internal parameters of the camera 2 were not corrected. 
[0156] In the above embodiment, description has been made 
of a case in which the optical condition to be varied in a 
camera of the type in which optical conditions can be 
varied is the focal length of a zoom camera. However, the 
present invention is not limited thereto. The condition to 
be varied in a camera may be the moving distance of the 
lens in an auto-focal camera or an internal parameter of a 
camera which are adjusted when the wavelength of light 
other than visible light such as infra-red rays, 
ultraviolet rays, X-rays or charged particle beam received 
by the camera as image data varies. 

[0157] In the above embodiment, a case where the image 
processing device has a stereo image producing part for 
performing three-dimensional measurement to produce 
orthoscopic images. However, the image data processing 
device of the present invention is not limited thereto. 
The image data processing device xnay perform three- 
dimensional measurement of an object directly without 
producing orthoscopic images. 

[0158] As has been described above, the device for 
measuring data for calibration of the present invention 
calculates data for calibration corresponding to the varied 
optical conditions of the camera in photographing using 
data for calibration calculated in the internal parameter 
calculating part and a plurality of optical conditions 
under which the images of the chart are photographed. 
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Thus, when images are photographed with a camera of the 
type in which optical conditions can be varied under varied 
conditions, data for calibration corresponding to the 
varied conditions in the camera in photographing necessary 
to produce a high-quality image free from the effect of the 
lens distortion can be easily obtained. 
[0159] The image data processing device of the present 
invention has an image data receiving part for receiving 
data of photographed images and optical photographing 
conditions from a camera of the type in which optical 
conditions can be varied, a calibration data forming part 
for forming data for calibration based on the optical 
photographing conditions received by the image data 
receiving part; and an image processing part for processing 
the data of photographed images based on the data for 
calibration. Thus, when images are photographed with a 
camera of the type in which optical conditions can be 
varied under varied conditions, high-quality images free 
from the effect of the lens distortion can be easily 
obtained using the data for calibration corresponding to 
the varied conditions in the camera in photographing. 
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